Please post the info requested in https://retropie.org.uk/forum/topic/3/read-this-first to your topic. Are you using RetroPie and/or the RetroPie's EmulationStation fork ?

I've tried setting gpu_mem to both 128 and 256, and I also commented out dtoverlay=vc4-kms-v3d.

The RetroPIe image doesn't have that overlay included - are you using the RetroPie image or not ?

Could the lack of a windowing system in OS Lite be causing this issue?

No, but please provide some context, as asked, on your setup.